COMMENT ⊗   VALID 00003 PAGES
C REC  PAGE   DESCRIPTION
C00001 00001
C00002 00002	.require "memo.pub[let,jmc]" source
C00036 00003	.single space
C00039 ENDMK
C⊗;
.require "memo.pub[let,jmc]" source;
.double space
.cb THE LITTLE THOUGHTS OF THINKING MACHINES

	When we interact with computers and other machines, we often use
language ordinarily used for talking about people.  We may say of a
vending machine, "It wants another ten cents for a lousy candy bar."
We may say of an automatic teller machine, "It thinks I don't have
enough money in my account because it doesn't yet know about the
deposit I made this morning." This article is about when we're right
or almost right in saying these things, and when it's a good idea to
think of machines that way.

	For more than a century we have used machines in our daily lives whose
detailed functioning most of us don't understand.  Few of us know much
about how the electric light system or the telephone system work
internally.  We do know their external behavior; we know that lights are
turned on and off by switches and how to dial telephone numbers.  We
may not know much about the internal combustion engine, but we know
that a car needs more gas when the gauge reads near EMPTY.

	In the next century we'll be increasingly faced with much more complex
computer based systems.  It won't be necessary for most people to
know very much about how they work internally, but what we will have
to know about them in order to use them is more complex than what we
need to know about electric lights and telephones.  As our daily lives
involve ever more sophisticated computers, we will find that ascribing
little thoughts to machines will be increasingly useful in understanding
how to get the most good out of them.

	Much that we'll need to know concerns the information stored
in computers, which is why we find ourselves using psychological
words like "knows", "thinks", and "wants" in referring to machines,
even though machines are very different from humans and these words
arose from the human need to talk about other humans.

	According to some authorities, to use these words, the language of
the mind, to talk about machines is to commit the error of
anthropomorphism.  Anthropomorphism is often an error, all right, but it
is going to be increasingly difficult to understand machines without
using mental terms.

	Ever since Descartes, philosophically minded people have wrestled
with the question of whether it is possible for machines to think.  As
we interact more and more with computers - both personal computers and
others - the questions of whether machines can think and what kind of
thoughts they can have become ever more pertinent.  We can ask whether
machines remember, believe, know, intend, like or dislike, want,
understand, promise, owe, have rights or duties, or deserve rewards
or punishment.  Is this an all-or-nothing question, or can we say that
some machines do some of these things and not others, or that they do
them to some extent?


	My answer is based on work in the field of artificial
intelligence (usually abbreviated AI)
 which is the science and engineering of making computers
solve problems and behave in ways generally considered to be intelligent.

	AI research usually involves programming
a computer to use specific concepts and to have specific mental
qualities.  Each step is difficult, and different programs have
different mental qualities.  Some programs acquire information from
people or other programs and plan actions for people that
involve what other people do.  Such programs must ascribe beliefs,
knowledge and goals to other programs and people.  Thinking about
when they should do so led to the considerations of this article.

	AI researchers now believe that much behavior can be
understood using the principle of rationality:

	%2It will do what it thinks will achieve its goals.%1

	What behavior is predicted then depends on
what goals and beliefs are ascribed.  The
goals themselves need not be justified as rational.

	Adopting this principle of rationality, we see that
different machines have intellectual qualities to
differing extents.  Even some very simple machines can be usefully regarded
as having some intellectual qualities.
Machines have and will have varied little minds.  Long before
we can make machines with human capability, we will have many machines
that cannot be understood except in mental terms.
  Machines can and will be
given more and more intellectual qualities; not even human intelligence
is a limit.  However, artificial intelligence is a difficult branch of
science and engineering, and, judging by present slow progress, it
might take a long time.  From the time of
Mendel's experiments with peas to the cracking of
 the DNA code for proteins,
a hundred years elapsed, and genetics isn't done yet.

	Present machines have almost no emotional qualities, and, in my
opinion, it would be a bad idea to give them any.  We have enough
trouble figuring out our duties to our fellow humans and to animals
without creating a bunch of robots with qualities that would allow
anyone to feel sorry for them or would allow them to feel sorry for
themselves.

	Since I advocate some anthropomorphism, I'd better explain what I
consider good and bad anthropomorphism.  Anthropomorphism is the 
ascription of human characteristics to things not human.  When is it
a good idea to do this?  When it says something that cannot as conveniently
be said some other way.

	Don't get me wrong.  The kind of anthropomorphism where someone
says, "This terminal hates me!" and bashes it, is just as silly as
ever.  It is also common to ascribe personalities to cars, boats,
and other machinery.  It is hard to say whether anyone takes this
seriously.  Anyway, I'm not supporting any of these things.  

	The reason for ascribing mental qualities and mental processes
to machines is the same as for ascribing them to other people.  It
helps understand what they will do, how our actions will affect
them, how to compare them with ourselves and how to design them.

	Researchers in artificial intelligence (AI) are interested in the
use of mental terms to describe machines for two reasons.  First we want
to provide machines with theories of knowledge and belief
so they can reason about what their users know, don't know, and
want.
  Second what the user knows about the machine can often best
be expressed using mental terms.

	Suppose I'm using an automatic teller machine at my bank.  I may
make statements about it like, "It won't give me any cash because it
knows there's no money in my account," or, "It knows who I am because
I gave it my secret number".  We need not acribe to the teller
machine the thought, "There's no money in his account," as its reason to
refuse to give me cash.  But it was designed to act as if it does,
and if I want to figure out how to make it give me cash in the future,
I should treat it as if it knows that sort of thing.

	It's difficult to be rigorous about whether a machine really
"knows", "thinks", etc., because we're hard put to define these
things.  We understand human mental processes only slightly better
than a fish understands swimming.

 	Current AI approaches to ascribing specific mental qualities
use the symbolism of mathematical logic.  In that symbolism,
speaking technically, 
a suitable collection of functions and predicates must be given.
Certain formulas of this
logic are then axioms giving relations between the concepts and
conditions for ascribing them.  These axioms are used by reasoning
programs as part of the process whereby the program decides what to
do.  The formalisms require too much explanation to be included in
this article, but some of the criteria are easily given in English.

	Beliefs and goals are ascribed in accordance with the
the principle of rationality.  Our object is to account
for as much behavior as possible by saying the machine or person
or animal does what it thinks will achieve its goals.  It is especially
important to have what is called in AI and %2epistemologically adequate%1
system.  Namely, the language must be able to express the information our
program can actually get about a person's or machine's "state of mind"
 - not just what might be obtainable if the
neurophysiology of the human or the design of the machine were more
accessible.

	In general we cannot give definitions, because the concepts
form a system that we fit as a whole to phenomena.  Similarly the
physicist doesn't give a definition of electron or quark.  Electron
and quark are terms in a complicated theory that predicts
the results of experiments.

	Indeed common sense psychology works in the same way.  A
child learns to ascribe wants and beliefs to others in a complex
way that he never learns to encapsulate in definitions.

	Nevertheless we can give approximate criteria for
some specific properties relating them to the more implicit
properties of believing and wanting.

%2Intends%1 - We say that a machine intends to do something if we
can regard it as believing that it will attempt to do it.  We may
know something that will deter it from making the attempt.
Like most mental concepts, intention is an intermediate in the causal
chain; an intention may be caused by a variety of stimuli and
predispositions and may result in action or be frustrated in a
variety of ways.

%2Tries%1 - This is important in understanding machines that have
a variety of ways of achieving a goal including possibly ways that
we don't know about.  If the machine may do something we don't know
about but that can later be explained in relation to a goal, we have
no choice but to use "is trying" or some synonym to explain the
behavior.

%2Likes%1 - As in "A likes B".  This involves a wanting  B's  welfare.
It requires that  A  be sophisticated enough to have a concept of
B's welfare.

%2Self-consciousness%1 -
Self consciousness is perhaps the most interesting mental quality to
humans.  Human self consciousness involves at least the following:

1. Facts about the person's body as a physical object.  This permits
reasoning from facts about bodies in general to one's own.
It also permits reasoning from facts  about one's own body, e.g.
its momentum, to corresponding facts about other physical objects.

2. The ability to observe one's own mental processes and to form
beliefs and wants about them.  A person can wish he were smarter
or didn't want a cigarette.

3. Facts about oneself as a having beliefs, wants, etc. among other
similar beings.

	Some of the above attributes of human self-consciousness
are easy to program.  For example, it is not hard to make a program
look at itself, and many AI programs do look at parts of themselves.
Others are more difficult.  Also animals cannot be shown to have
more than a few.  Therefore, many present and future programs can
best be described as partially self-conscious.

	Suppose someone says, "The dog wants to go out".  He has ascribed
the mental quality of wanting to the dog without claiming that the dog
thinks like a human and can form out of its parts the thought, "I want
to go out".

	The statement isn't shorthand for something the dog did, because there
are many ways of knowing that a dog wants to go out.  It also isn't
shorthand for a specific prediction of what the dog is likely to
do next.  Nor do we know enough about the physiology of dogs for
it to be an abbreviation for some statement about the dog's nervous
system.  It is useful because of its connection with all of these
things and because what it says about the dog corresponds in
an informative way with similar statements about people.  It doesn't
commit the person who said it to an elaborate view of the mind
of a dog.  For example, it doesn't commit a person to any position
about whether the dog has the mental machinery to know that it is
a dog or even to know that it wants to go out.
We can make similar statements about machines.

	Here is an extract from the instructions that came with
an electric blanket.  %2"Place the control near the bed in a place that
is neither hotter nor colder than the room itself.  If the control
is placed on a radiator or radiant heated floors, it will "think"
the entire room is hot and will lower your blanket temperature, making
your bed too cold.  If the control is placed on the window sill
in a cold draft, it will "think" the entire room is cold and will
heat up your bed so it will be too hot."%1

	I suppose some philosophers, psychologists, and English teachers
would maintain that the blanket manufacturer is guilty of anthropomorphism
and some will claim that great harm can come from thus ascribing
to machines qualities which only humans can have.  I argue
that saying that the blanket thermostat "thinks" is ok; they could
even have left off the quotes.  Moreover, this helps us understand
how the thermostat works.  The example is extreme, because most
people don't need the word "think" to understand how a thermostatic
control works.  Nevertheless, the blanket manufacturer was probably
right in thinking that it would help some users.

	Keep in mind that the thermostat can only be properly considered
to have just three possible thoughts or beliefs.  It may believe that
the room is too hot, or that it is too cold, or that it is ok.  It
has no other beliefs; for example, it does not believe that it is
a thermostat.

	The example of the thermostat is a very simple one.  If we had only
thermostats to think about, we wouldn't bother with the concept of
belief at all.  And if all we wanted to think about were zero and one,
we wouldn't bother with the concept of number.

	Here's a somewhat fanciful example of a machine that might
someday be encountered in daily life with more substantial mental
qualities.

	In ten or twenty years Minneapolis-Honeywell, which makes
many thermostats today, may try to sell you a really fancy home temperature
control system.  It will know the preferences of temperature and
humidity of each member of the family and can detect who is in the
room.  When several are in the room it makes what it considers a
compromise adjustment taking account who has most recently had
to suffer having the room climate different from what he
prefers.  Perhaps Honeywell discovers that these compromises should be
modified according to a social rank formula devised by its psychologists
and determined by patterns of speech loudness.  The brochure
describing how the thing works is rather lengthy and the real dope
is in a rather technical appendix in small print.

	Now imagine that I went on about this thermostat until you
were bored and you skipped the rest of the paragraph.  Confronted
with an uncomfortable room you form any of the following hypotheses
depending on what other information you had.

1. It's trying to do the right thing, but it can't because the valve
is stuck.  But then it should complain.

2. It regards Grandpa as more important than me, and it is keeping
the room hot in case he comes in.

3. It confuses me with Grandpa.

4. It has forgotten what climate I like.

	A child unable to read the appendix to the user's manual
will be able to understand a description of the "climate controller"
in mental terms.  The child will be able to request changes like
%2"Tell it I like it hotter"%2 or %2"Tell it Grandpa's not here now%1.
  Indeed the designer of the system will have used
the mental terms in formulating the design specifications.

	The automatic teller is another example.  It
has beliefs like, "There's enough money in the account," and "I don't
give out that much money".  A more elaborate automatic
teller that handles loans, loan payments, traveler's checks,
and so forth, may have beliefs like, "The payment wasn't made on time,"
or, "This person is a good credit risk."

	The next example is adapted from the University of California
philosopher John Searle.  A person
who doesn't know Chinese memorizes a book of rules for manipulating Chinese
characters.  The rules tell him how to extract certain parts of a sequence
of characters, how to re-arrange them, and how finally to send back
another sequence of characters.  These rules say nothing about the meaning
of the characters, just how to compute with them.
 He is repeatedly given Chinese sentences, to which he applies the rules, and
gives back what turn out, because of the clever rules, to be Chinese sentences
that are appropriate replies.
We suppose that the rules result in a
Chinese conversation so intelligent that the person giving and receiving
the sentences can't tell him from an intelligent Chinese.
This is analogous to a computer, which only obeys its
programming language, but can be programmed such that one can communicate
with it in a different programming language, or in English.
Searle says that since the person in the example doesn't understand
Chinese - even though he can produce intelligent Chinese conversation
by following rules - a computer cannot be said to "understand" things.
He makes no distinction, however, between the hardware (the person)
and the process (the set of rules).  I would argue that the set of
rules understands Chinese, and, analogously, a computer program may
be said to understand things, even if the computer does not.
Both Searle and I are ignoring practical difficulties like how long
it would take a person with a rule book to come up with a reply.

	Daniel Dennett, Tufts University philosopher, has proposed three
attitudes aimed at understanding a system with which one interacts.

	The first he calls the physical stance.  In this we look at the
system in terms of its physical structure at various levels of organization.
Its parts have their properties and they interact in ways
that we know about.  In principle the analysis can go down to the atoms
and their parts.  Looking at a thermostat from this point of view,
we'd want to understand the working of the bimetal strip that most
thermostats use.  For the automatic teller, we'd want to know about
integrated circuitry, for one thing.  (Let's hope no one's in line
behind us while we do this).

	The second is called the design stance.  In this we analyze something
in terms of the purpose for which it is designed.  Dennett's example of
this is the alarm clock.  We can usually figure out what an alarm
clock will do, e.g.  when it will go off, without knowing whether it
is made of springs and gears or of inegrated circuits.  The user of
alarm clock typically doesn't know or care much about its internal
structure, and this information wouldn't be of much use.  Notice that
when an alarm clock breaks, its repair requires taking the physical
stance.  The design stance can usefully be applied to a thermostat -
it shouldn't be too hard to figure out how to set it, no matter how
it works.  With the automatic teller, things are a little less clear.

	The design stance is appropriate not only for machinery but also for
the parts of an organism.  It is amusing that we can't attribute a purpose
for the existence of ants, but we can find a purpose for the
glands in an ant that emit a chemical substance for other ants to follow.

	The third is called the intentional stance, and this is what we'll
often need for understanding computer programs.  In this we try to
understand the behavior of a system by ascribing to it beliefs, goals,
intentions, likes and dislikes, and other mental qualities.  In this
stance we ask ourselves what the thermostat thinks is going on, what
the automatic teller wants from us before it'll give us cash.  We say
things like, %2"The store's billing computer wants me to pay up, so it
intends to frighten me by sending me threatening letters"%1.  The intentional
stance is most useful when it is the only way of expressing what we know
about a system.

	(For variety Dennett mentions the astrological stance.  In this the
way to think about the future of a human is to pay attention to the
configuration of the stars when he was born.  To determine whether an
enterprise will succeed we determine whether the signs are favorable.
This stance is clearly distinct from the others - and worthless.)

	It is easiest to understand the ascription of thoughts
to machines in circumstances when we also understand the machine
in physical terms.  However, the payoff comes when either no-one
or only an expert understands the machine physically.

	However, we must be careful not to ascribe properties to a
machine that the particular machine doesn't have.  We
humans can easily
fool ourselves when there is something we want to believe.

	The mental qualities of present machines are not the same as ours.
While we will probably be able, in the future, to make machines with
mental qualities more like our own, we'll probably never want to deal with
machines that are too much like us.  Who wants to deal with a computer that
loses its temper, or an automatic teller that falls in love? Computers
will end up the the psychology that is convenient to their designers -
(and they'll be fascist bastards if those designers don't think twice).
Program designers have a tendency to think of the users as idiots who need
to be controlled.  They should rather
think of their program as a servant, whose
master, the user, should be able to control it.  If designers and
programmers think about the apparent mental qualities that their
programs will have, they'll create programs that are easier and pleasanter
- more humane -
to deal with.

.single space
References:

%3Dennett, Daniel (1981)%1: "True Believers: the Intentional Strategy and Why it
Works," %2The Herbert Spencer Lectures%1, A. Heath (ed.),   
Oxford University Press, 1981.
This non-technical article describes the physical, design and intentional
stances in philosophical language.

%3Kowalski, Robert (1979)%1: %2Logic for Problem Solving%1, North Holland
Elsevier, New York, 1979.
This book describes the use of logical formalism in artificial intelligence.

%3McCarthy, John (1979)%1:
"Ascribing Mental Qualities to Machines" in %2Philosophical Perspectives 
in Artificial Intelligence%1, Ringle, Martin (ed.), Harvester Press, July 1979.
This is the technical paper on which this article is based.

%3McCarthy, John (1979)%1: 
"First Order Theories of Individual Concepts and Propositions", 
in Michie, Donald (ed.) %2Machine Intelligence 9%1, (University of
Edinburgh Press, Edinburgh).
This paper uses the mathematical formalism of first order logic
to express facts about knowledge.

%3Newell, Allen (1980)%1:  "The Knowledge Level," %2 Artificial Intelligence%1,
Vol. 18 No.1, pp. 87 - 127, January 1982.
This article clearly expounds a different approach to ascribing
mental qualities.

%3Searle, John (1980)%1: "Minds, Brains and Programs," %2Behavioral and
Brain Sciences%1, Vol.3 No. 3, pp. 417-424, September 1980.
This article takes the point of view that mental qualities should not
be ascribed to machines.